A [[Sparsity x LLMs]] approach from the HazyResearch lab at Stanford. - The Pixelated Butterfly method **operates at the level of matrix operations**, which are fundamental to many AI models. Matrix operations involve multiplying matrices, which are arrays of numbers, to perform computations. - The authors of the Pixelated Butterfly method combine **specialized butterfly** and **low-rank matrices** to create a simple and efficient sparse training method. - Butterfly matrices are a type of matrix that can be used to **approximate other matrices**, - Low-rank matrices have a small number of non-zero elements. - By using butterfly and low-rank matrices, the Pixelated Butterfly method can effectively **reduce the number of non-zero elements in a matrix**, which in turn reduces the computational complexity of the AI model. - The Pixelated Butterfly method **can be applied to most major network layers** that rely on matrix multiplication, which includes many popular types of neural networks. - Overall, the Pix      elated Butterfly method is an example of how sparsity techniques can be used to simplify AI models and make them more efficient. ![[Pasted image 20230801120624.png]] ![[Pasted image 20230801120500.png]] Ref: 1. https://hazyresearch.stanford.edu/blog/2022-01-17-Sparsity-3-Pixelated-Butterfly 2. https://arxiv.org/abs/2112.00029